fix(sync): bootstrap cursorless channels in latest-only mode by obviyus · Pull Request #182 · openclaw/discrawl

obviyus · 2026-08-19T13:29:22Z

Problem

Routine and scheduled --latest-only syncs discover new active channels and threads, but skip their messages forever because those surfaces do not yet have a latest_message_id cursor. Channel metadata is published while the message archive remains empty.

Fix

Fetch at most one newest page when latest-only sync encounters a cursorless channel.
Persist the newest message cursor and oldest fetched backfill boundary.
Leave longer history incomplete so a later --full run resumes from that boundary.
Reuse the canonical backfill loop; no thread-specific behavior.
Align the routine-sync documentation; leave release-owned changelog curation untouched.

Real Discord transport proof

Ran this branch with a real Discord bot against a brand-new disposable config and SQLite archive. Guild/channel IDs and credentials are redacted; no message content was inspected or retained.

$ discrawl sync --source discord --guild <redacted> --skip-members --latest-only
message sync started ... channels=6 full=false
message sync finished ... messages_written=261 ... elapsed=2s
{guilds:1, channels:14, threads:0, members:0, messages:261}

$ sqlite3 <disposable-db> <cursor-state-query>
kind=text stored_count=100 latest_cursor=set backfill_cursor=set history_complete=""

$ discrawl sync --source discord --guild <redacted> --channels <redacted> --full --since 2022-01-01T00:00:00Z --skip-members
message sync finished ... messages_written=34 ... elapsed=3s
kind=text stored_count=134 oldest_time=2022-01-17T15:06:21Z latest_cursor=set backfill_cursor=set history_complete=""

$ discrawl sync --source discord --guild <redacted> --channels <redacted> --full --skip-members
message sync finished ... messages_written=3578 ... elapsed=35s
kind=text stored_count=3712 oldest_time=2018-07-21T13:13:45Z latest_cursor=set backfill_cursor=set history_complete=1

This proves the production Discord client fetched one bounded head page for a cursorless channel, persisted both cursor boundaries, resumed older history from that boundary, and completed the archive on a later full pass. The disposable archive was deleted afterward.

Automated proof

Regression test reproduces an active forum thread with 101 messages. Latest-only stores exactly 100 with one API call; full sync later stores the remaining message and marks history complete.
GOWORK=off go test ./...
make lint
make test-race
make test-coverage (85.1%)
make smoke
Autoreview: clean, no actionable findings.

clawsweeper · 2026-08-19T13:29:25Z

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

clawsweeper · 2026-08-19T13:36:57Z

Codex review: needs maintainer review before merge. Reviewed August 19, 2026, 10:43 AM ET / 14:43 UTC.

ClawSweeper review

What this changes

The PR makes latest-only Discord archive sync fetch one newest page for a newly discovered channel or thread without a cursor, then retain cursor state for a later full backfill.

Merge readiness

⚠️ Ready for maintainer review - 1 item remains

Keep open: current main still skips cursorless channels in latest-only mode, while this PR adds the bounded bootstrap and has corrected the previously missed guide. No source-level correctness blocker was found; wait for the in-progress test check before merge.

Priority: P2
Reviewed head: ff97f331b6bfa3a97cda819ee4c5068806d009bd

Review scores

Measure	Result	What it means
Overall readiness	🦞 diamond lobster (5/6)	A focused implementation with direct production-transport evidence, stateful regression coverage, and no actionable review finding.
Proof confidence	🦞 diamond lobster (5/6)	Sufficient (terminal): Redacted terminal output from a real Discord bot and SQLite archive shows bounded bootstrap, persisted cursors, resumed backfill, and eventual completion.
Patch quality	🦞 diamond lobster (5/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Verified	Sufficient (terminal): Redacted terminal output from a real Discord bot and SQLite archive shows bounded bootstrap, persisted cursors, resumed backfill, and eventual completion.
Evidence reviewed	6 items	Current-main defect: Current main returns before requesting messages whenever latest-only mode finds no stored latest cursor, which matches the reported cursorless-channel failure. Bounded implementation: The PR routes a cursorless latest-only channel through the existing backfill paginator with a one-page limit, then advances the latest cursor. Regression coverage: The new forum-thread regression verifies one 100-message page and one API call, preserves an incomplete backfill cursor, then completes the remaining history in a later full sync.
Findings	None	None.
Security	None	None.

Live Verification

Command: go run ./cmd/discrawl sync --help

Result: FAIL (failed) — execution before step 1 run: sh -lc pnpm install --ignore-scripts --frozen-lockfile failed: ! Corepack is about to download https://registry.npmjs.org/pnpm/-/pnpm-11.22.0.tgz

sh -lc pnpm install --ignore-scripts --frozen-lockfile failed: ! Corepack is about to download https://registry.npmjs.org/pnpm/-/pnpm-11.22.0.tgz

Assertions:

FAIL expect_output: --latest-only

How this fits together

Discord sync discovers accessible channels and threads, then writes messages and per-channel cursors to the local SQLite archive. Latest-only runs update the newest available data, while full runs continue from the stored backfill boundary.

flowchart LR
A[Discord channels and threads] --> B[Sync mode selection]
B --> C[Stored cursor check]
C --> D[One-page latest bootstrap]
C --> E[Incremental or full backfill]
D --> F[SQLite messages and cursors]
E --> F

Before merge

Complete next step (P2) - No mechanical repair is identified on the current head; this PR awaits ordinary maintainer review and completion of its running test check.

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Changed surface	4 files: 1 production, 1 test, 2 docs	The implementation remains focused on the message-sync path and its mirrored operator guidance.
LOC delta	production +26/-5; tests +48/-15; docs +2/-2	The production growth is limited to a bounded reuse of the existing backfill path and is covered by a stateful regression.

Technical review

Best possible solution:

Merge the bounded bootstrap after required checks complete so newly discovered active surfaces receive a current message page and a later full sync can finish their history.

Do we have a high-confidence way to reproduce the issue?

Yes—source-reproducible with high confidence: current main returns immediately for latest-only channels without a stored cursor, and the PR regression exercises the resulting forum-thread path.

Is this the best way to solve the issue?

Yes. Reusing the established backfill paginator with a one-page cap preserves cursor semantics and avoids a separate thread-specific synchronization path.

AGENTS.md: found, but no applicable review policy affected this item.

Codex review notes: model internal, reasoning high; reviewed against 679ed223ab13.

Labels

Label changes:

add rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
remove rating: 🐚 platinum hermit: Current PR rating is rating: 🦞 diamond lobster, so this older rating label is no longer current.

Label justifications:

P2: This repairs routine archive synchronization for newly discovered Discord channels and threads with a bounded user impact.
rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): Redacted terminal output from a real Discord bot and SQLite archive shows bounded bootstrap, persisted cursors, resumed backfill, and eventual completion.
proof: sufficient: Contributor real behavior proof is sufficient. Redacted terminal output from a real Discord bot and SQLite archive shows bounded bootstrap, persisted cursors, resumed backfill, and eventual completion.

Evidence

What I checked:

Current-main defect: Current main returns before requesting messages whenever latest-only mode finds no stored latest cursor, which matches the reported cursorless-channel failure. (internal/syncer/message_sync.go:236, 679ed223ab13)
Bounded implementation: The PR routes a cursorless latest-only channel through the existing backfill paginator with a one-page limit, then advances the latest cursor. (internal/syncer/message_sync.go:239, ff97f331b6bf)
Regression coverage: The new forum-thread regression verifies one 100-message page and one API call, preserves an incomplete backfill cursor, then completes the remaining history in a later full sync. (internal/syncer/syncer_test.go:593, ff97f331b6bf)
Prior review finding resolved: The matching public sync-sources guide now describes cursorless one-page bootstrap, removing the contradiction identified in the prior review. (docs/guides/sync-sources.md:21, ff97f331b6bf)
Feature history: Latest-only mode originated in Peter Steinberger's commit, and Hannes Rudolph later hardened monotonic latest cursor handling in the same sync path. (internal/syncer/message_sync.go:222, e1dcd1ea4902)
Real transport proof: The PR body supplies redacted terminal output from a real Discord bot and SQLite archive: a bounded latest-only fetch, persisted latest/backfill cursors, then partial and complete full backfills. (ff97f331b6bf)

Likely related people:

Peter Steinberger: Introduced the original latest-only mode and remains the primary current-main history signal for this sync path. (role: introduced latest-only sync behavior; confidence: high; commits: e1dcd1ea4902, 089c1f5b6ae4; files: internal/syncer/message_sync.go, internal/syncer/syncer_test.go)
Hannes Rudolph: Authored the monotonic latest-checkpoint repair in the same message synchronization code. (role: recent cursor-state contributor; confidence: high; commits: 265a76cb90e3; files: internal/syncer/message_sync.go)

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (2 earlier review cycles)

reviewed 2026-08-19T13:32:35.980Z sha 84fa9b7 :: needs real behavior proof before merge. :: [P3] Update the routine-sync documentation | [P3] Remove the release-owned changelog entry
reviewed 2026-08-19T14:34:07.693Z sha c730a64 :: needs changes before merge. :: [P3] Update the matching sync-sources guide

obviyus · 2026-08-19T14:58:15Z

@clawsweeper automerge

clawsweeper · 2026-08-19T14:58:18Z

🦞👀
ClawSweeper picked this up.

Command router queued. I will update this comment with the next step.

fix(sync): bootstrap cursorless channels in latest-only mode

84fa9b7

docs(sync): align latest-only behavior

c730a64

docs(sync): align source guide with bootstrap behavior

ff97f33

clawsweeper Bot added rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. labels Aug 19, 2026

obviyus merged commit 5a0dc6f into openclaw:main Aug 19, 2026
13 checks passed

Navigation Menu

Uh oh!

fix(sync): bootstrap cursorless channels in latest-only mode - #182

fix(sync): bootstrap cursorless channels in latest-only mode#182
obviyus merged 3 commits into
openclaw:mainfrom
obviyus:fix/bootstrap-cursorless-channels

obviyus commented Aug 19, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Aug 19, 2026

Uh oh!

clawsweeper Bot commented Aug 19, 2026 •

edited

Loading

Security

Review metrics

Technical review

Labels

Evidence

Rating scale

Workflow

History

Uh oh!

obviyus commented Aug 19, 2026

Uh oh!

clawsweeper Bot commented Aug 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

obviyus commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Fix

Real Discord transport proof

Automated proof

Uh oh!

clawsweeper Bot commented Aug 19, 2026

Uh oh!

clawsweeper Bot commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ClawSweeper review

What this changes

Merge readiness

Review scores

Verification

Live Verification

How this fits together

Before merge

Security

Review metrics

Technical review

Labels

Evidence

Rating scale

Workflow

History

Uh oh!

obviyus commented Aug 19, 2026

Uh oh!

clawsweeper Bot commented Aug 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

obviyus commented Aug 19, 2026 •

edited

Loading

clawsweeper Bot commented Aug 19, 2026 •

edited

Loading